Biological Principles in Self-Organization of Young Brain - Viewed from Kohonen Model 
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Variants of the Kohonen model are proposed to study biological principles of self-organization in a model 
of young brain. We suggest a function to measure aquired knowledge and use it to auto-adapt the topology of 
neuronal connectivity, yielding substantial organizational improvement relative to the standard model. In the 
early phase of organization with most intense learning, we observe that neural connectivity is of Small World 
type, which is very efficient to organize neurons in response to stimuli. In analogy to human brain where pruning 
of neural connectivity (and neuron cell death) occurs in early life, this feature is present also in our model, which 
is found to stabilize neuronal response to stimuli. 

PACS numbers: 87.10.+e,87.19.La,89.75.Fb,87.19.Dd 



Introduction. In physics the reductionist school of 
thought has long time prevailed, which postulates that 
the way of understanding laws of nature is by looking at 
its most elementary building blocks. This point of view 
has been questioned from the viewpoint of complexity, 
which says that in nature there are emergent phenom- 
ena which cannot be predicted from the laws of ele- 
mentary constituents - for example the shape of a snow 
flake cannot be obtained from the laws of hydrogen and 
oxygen atoms |lj. A prime example of complexity is 
the self-organization of human brain. Experimental in- 
formation on the organization of brain comes from ob- 
servation of power laws 1 //" (with a ~ 1) in the fre- 
quency power spectrum of electrical signals measured 
by electroencephalograms (EEG) [51 and by magne- 
toencephalograms (MEG) |4J]. Such power laws reflect 
a fractal temporal behavior |5|]. Power laws have been 
found also in size distributions of epilectic seizures and 
neural avalanches q], hinting to a fractal behavior in 
seize. To explain such power laws the sand pile model of 
self-organized criticality (SOC) has been devised |H 
and used to model EEG spectra |')| . On the other hand, 
power laws in frequency have been observed in local 
field potentials (LFP) measured in cat parietal associ- 
ation cortex, which seem inconsistent with the SOC 
model 1 10]. From the information theory point of view, 
however, it is not clear which information is transmitted 
in EEG signals or neural avalanches. 

Here we consider self-organization of young brain, 
based on the Kohonen model It is a biolog- 

ical property that the organization of visual cortex in 
mammals occurs during a relative short time window 
shortly after birth during which neural connectivity is 
established (3 [3- Most remarkably, this is accom- 



panied by genetically controlled neuron cell death and 
pruning of synaptic connections In brains of 

mature adults, functional magnetic resonance imaging 
(FMRI) has shown evidence for functional networks 
of correlated brain activity fl7ll . which display small 
world and scale free architecture. Small World Net- 
work (SWN) neural connectivity has been found in a 
network of Hodgkin-Huxley neurons to give a fast and 
coherent system response 11811 . in an associative mem- 
ory model to improve restoration of perturbed memory 
patterns fpjll and to reduce learning time and error in a 
multi-layer learning model i2oll . 

In this paper, three main issues are addressed: (i) we 
establish a link between network topology and flow of 
information, and show that biologically inspired auto- 
adaption of the Kohonen network improves organiza- 
tion; (ii) we observe that SWN topology is present dur- 
ing most of the organizational phase; (iii) we show that 
biologically observed pruning of synaptic connections 
in the early evolution of the network is beneficial for or- 
ganization, while reconnections give an adverse effect. 
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Kohonen model. The Kohonen network II 111 is a 
model of organization of information in the visual cor- 
tex in the brain. Information coming from the eye may 
consist of luminosity/darkness, directions/shapes, col- 
ors, motions in all directons of the visual field, repre- 
sented by a large set of stimuli in a high-dimensional 
space. One of the brain tasks is to recognize images 
and group them into classes like, e.g., humans, animals, 
buildings, roads, etc. In the Kohonen model, this is 
achieved by reducing (mapping) the high-dimensional 
manifold of Q stimuli to a low-dimensional (2-D) man- 
ifold, represented by N <C Q neurons, which maintain 
topological continuity, i.e. proximate stimuli yield prox- 
imate neurons. The dynamical rules of the standard 
Kohonen map are as follows There are stimuli 

q = 1 , . . . , Q, located at p q in a vector space, and there 
are neurons, i = 1, . . . ,/V, living on a 2-D grid map (Fig. 
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FIG. 1: Initial regular grid map of neuron topology (neigh- 
bors): V g = 1 (left) and V g = 3 (right). 
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FIG. 2: Standard Kohonen. Relation between neighborhood 
order V(t) and total number of connexions M(t). 



[0. To each neuron is assigned a weight vector iv,- living 
in the same vector space as the stimuli. At each time 
/, a stimulus p q is randomly selected, and the neuronal 
weights are updated according to a rule derived from 
unsupervised Hebbian learning: 

m(t+i)=w i (t)+^t)\p q (t)-w i (t)],Viev g (t), (i) 

where neuron g is the "winning" neuron, being closest 
to stimulus pq, and V g defines an order of neighborhood 
(topology) for neuron g, given by a regular grid map 
(Fig. [0. For example, V g = 1 corresponds to the 4 near- 
est neighbors of g, while V g = 3 includes the 20 next-to- 
nearest neighbors. The parameter r| denotes the learning 
rate. Using this rule, the neuronal map learns the topol- 
ogy of the data set and eventually becomes deformed 
(for example, see Fig. [8}. The order of neighborhood 
V(t) is initially high, i.e. the map is highly connected. 
During evolution of organization V (t ) decreases gradu- 
ally, as shown in Fig. |2] r|(f) also decreases linearly in 
time to ensure convergence of neuronal weights. 

Knowledge and auto-adaptivity. In the standard Ko- 
honen model, the evolution of the map topology V(f) 
as well as the learning rate r|(f ) are determined a priori. 
This conditions the final absolute error of modelization, 
E ext , given by 
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(2) 



Fig. [3] shows a typical evolution of E ext over time (V 
goes in steps from 3 to 1). Guided by the biological 
principle of adaptation and plasticity of the brain, we 
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FIG. 3: Standard Kohonen. Temporal evolution of absolute 
error E ext and neighborhood order V. 



suggest to adapt dynamically V(t) and T|(/) to the state 
of the network at time t, and in particular to the cumula- 
tively learnt information. Moreover, we suggest to intro- 
duce a locally independent learning rate T|,-(r) for each 
neuron. For such purpose we construct (inspired by the 
Growing Neural Gas network EH ) the function of local 
attraction A'"' (f), 

A i r(t)=AT t (t-l)+8 itg{t) \\p q (t)-w i (t)\\ , (3) 

for all 2 = 1,. . . ,N, with initial value Af(0) = 0. This 
function counts how much the weight vector of a neuron 
i has moved, being an indicator how much that neuron 
has learnt. Low/high attraction means that such neuron 
has rarely/often been a "winner". From local attraction, 
we construct an adapted learning rate individually dif- 
ferent for each neuron (which is biologically more plau- 
sible) of the following form 



i 



Af(t) 



(4) 



The learning rate is chosen to be inversely proportional 
to local attraction, because for a neuron not having 
learnt at all it is advantageous to have a high potential 
learning speed, while a neuron which has learnt much 
tends to saturation, i.e. learning speed becomes slow. In 
this way we obtain a new version - called multi-rhythm 
- of the Kohonen model. 

Starting from such new learning rate, we will define 
an internal knowledge function K"". Intuitively, one ex- 
pects that learning rates are low and homogeneously 
distributed among the neurons, when the network has 
achieved organization, i.e. has learnt much informa- 
tion. This is the basis for the following definition of 
the knowledge function K mt , expressed in terms of an 
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FIG. 4: Self-instructed multi-rhythm model. Correspondence 
between K"" and E ext during the learning process. 

harmonic mean, 

K m, {t) = - = 1, (5) 

£ A'«<(t) 

with initial value K'"'(0) = and where A'"' denotes 
the average of A'"' over all neurons i £ { 1 , . . . ,N}. The 
function K m varies in the range 1 /N < K"" < 1, and be- 
comes maximal when all A" 1 ' are equal. Fig.0]shows the 
temporal behavior of function K mt and the absolute er- 
ror E ext . The decreasing error function E ext mirrors the 
behavior of function K" 1 ' and hence justifies the inter- 
pretation of the latter as knowledge gain. Fig. [5] shows 
the complementary behavior of E ext and K"" for the spe- 
cial case where V(t) is kept constant during all of the 
learning process. It should be noted that although er- 
ror E exl can be used to measure the error at the end of 
organization, it is not part of the Kohonen network, i.e. 
not available dynamically. However, K int (t) is available 
to the network at any time t. Thus we propose to dy- 
namically adapt the number of connections M{t) using 
K""(t): 

M(t) - -K int {t) . (6) 

This yields another version - called self-instructed - 
of the Kohonen model, where the change of topology 
adapts itself to the rhythm of knowledge gain. 

Dynamical link between topology and information 
content. In information theory, Marchiori and Latora 
1E2I E3I1 defined network connectivity dimensions on 
global and local scales, D g i b a i and D\ oca i, which ex- 
plicitely show the link between topology and function 
of the network, which here means efficient transmis- 
sion of information. A network of SWN architecture 
is both locally and globally efficient, corresponding to 
D global and Di oca i being low 13 Ej- Fig. |6] shows 
the correspondence between D g i uaa i, Di oca i, and abso- 
lute error E ext . In biology, often there is a link be- 
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FIG. 5: Self-instructed multi-rhythm model. Correspondence 
between K'"' and E exl for given order of neighborhood. 
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FIG. 6: Self-instructed multi-rhythm model. Global and lo- 
cal connectivity dimension D g i i, a i (open circles), -D; OCI/ / (dots) 
and absolute error E ext (asterix) during evolution of organiza- 
tion. 

tween structure and function, e.g. the structure of the 
hemoglobin molecule fits its functional task of trans- 
porting oxygen. Fig. |6] shows that D g i oba i and Di ocal 
are both low - indicating SWN connectivity, i.e. the net- 
work is highly connected at all length scales. SWN con- 
nectivity has been found beneficial in supervised learn- 
ing I20II . Here, SWN topology persists during most of 
the organizational phase, i.e. the regime where the er- 
ror decreases until saturation. This regime corresponds 
to an increase of accumulated (learnt) knowledge (see 
Fig. |4}. Towards the end of organization, the network 
looses its SWN character (refinement of spatial scale of 
learning, formation of separate islands of neurons rep- 
resenting classes of stimuli). The pruning of connec- 
tions appears to be an advantage because the progres- 
sive independence of neurons leads to a better precision 
in the local placement of neural weights. This has been 
confirmed by exploring the alternative of allowing for 
reconnections, which led to a larger error and deterio- 
ration of organization (see Fig. 0. This gives a pos- 
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TABLE I: Comparison between standard and self-instructed 
multi-rhythm model. Absolute error and standard deviation 
over 10 replications in parentheses. 



0.2 0.4 0.6 0.! 
Ratio of cut links 



Data set 


Eext(x) 

standard 


Eext(t) 

self-instructed 


Eext ( ^ ) self —instructed 


Eext{T)standard 


1 


0.446(0.023) 


0.222(0.006) 


0.498(0.013) 


2 


1.589(0.029) 


1.055(0.014) 


0.664(0.009) 


3 


2.302(0.458) 


0.442(0.030) 


0.192(0.013) 


Pima 


5.03(0.37) 10 5 


2.33(0.07) 10 5 


0.462(0.014) 


H. Fame 


2.06(0. 12) 10 5 


1.156(0.02) 10 s 


0.562(0.009) 



FIG. 7: Multi-rhythm model. Comparison of error for cutting 
links with and without reconnections. Links cut from the reg- 
ular grid. Results correspond to data set 2, replicating the test 
10 times, but are almost identical for all data sets. 



sible explanation for biologically observed pruning of 
synaptic connections during organization in the brain of 
mammals in a relatively short postnatal period. 

Results. For a fixed learning duration (x = 10 4 iter- 
ations until stability of organization has been reached), 
the standard Kohonen model is compared with the self- 
instructed multi -rhythm version, using a 10 x 10 neu- 
ral grid map. First, three different 2-D data sets are 
tested. They are each composed of Q — 800 stimuli 
with different spatial distributions (see Fig. [8}. Fig|8] 
shows that the neurons are closer to stimuli in the self- 
instructed multi-rhythm version relative to the standard 
version. Also a substantial reduction in the final abso- 
lute error E ext (i) is observed (see TabUJ. Ranging be- 
tween only 19% to 66% of the error of the standard Ko- 
honen model, the self-instructed multi-rhythm version 
is shown to be both more efficient and robust. Indeed, 
the standard deviations obtained over 10 replications is 
very low (w 1%). Second, some high dimensional data 
sets were also tested. Among them, the "Pima Indians 
Diabetes", 8-dimensions with Q = 768 stimuli |24|, and 
the "Hall of Fame", 15-dimensions with 0= 1320 El. 
and found qualitatively similar results (see Tab. GJ. 

In summary, a dynamical link between geometry and 
function (organization) was introduced in the Koho- 
nen network, based on its dynamically aquired infor- 
mation. Thus adaptation, also found in nature, leads 
to substantial improvement of organization in the con- 
text of our model. At the beginning of organization, the 
SWN topology causes communication at all scales, both 
global and local, and the network learns very quickly. 
We postulate that such topology is beneficial in any in- 
formation organizing network. As knowledge increases, 
the communication scale changes towards more local 
connectivity, until total independence of neurons due to 
pruning is reached. We found pruning of connections 
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FIG. 8: Comparison between standard (left column) and self- 
instructed multi-rhythm (right column) map, showing stim- 
uli (light crosses) and neurons (bold dots) for 2-D data sets 
(top=l, middle=2, bottom=3). Lines represent the deformed 
map at the end of organization. 



(in contrast to reconnections) to play a beneficial role in 
this model of self-organization of the visual cortex. 
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